Analysis of different model selection criteria

In this report we analyze different model selection criteria that were used in statistics, reconstruction, recognition and other areas. We concentrate on criteria based on Bayesian theory, MDL theory, and predicting future residuals. Our analysis showed that these criteria often are not consistent when the number of points in the underlying data of the model is changed or if the scale of the data is changed. At the end we show experiments, where it can be seen that all criteria are more or less equivalent when we have the appropriate models to model or explain the data. If not, then there are differences between the criteria but in general it is not possible to judge which one is superior to the others. 1 Related work Several model selection criteria were introduced over the past years. They are usually based on the following principles: • Bayesian rule, • MDL principle, • prediction of future residual, • hypothesis testing, • heuristic approaches. 1.1 Model selection based on Bayesian rule Model selection criteria based on Bayesian rule usually try to maximize the probability P (D/m, I) = ∫ ∫ L(θm)P (θm/m, I)P (σ/I)dθmdσ. (1) In the equation above, D denotes data points, m a type of model, I some prior information, θm parameters of model m and σ is a variance of noise in the data. L(θm) is the likelihood of the model m to the data D and can be approximated by the exponential of negative sum of squared distances of the data points to the model m when the noise distribution in the data is N(0, σ2). By P (θm/m, I) we denote the prior probability of θm, which is usually not known. Thus the uniform probability distribution of θm is often